Relevance & Research Question: In Web surveys, rating scales are typically presented in grid (or matrix) questions. Besides benefits such as the neat arrangement and efficient processing of rating scale items, grid questions carry an increased risk of respondents relying on cognitive shortcuts in order to reduce their cognitive and navigational effort. Even though in Web surveys a wide range of visual and dynamic features is available for the design of survey questions, new types of rating scale designs beyond grid questions using conventional radio buttons have rarely been used yet. In this study, two different rating scale procedures using drag-and-drop as a more interactive data input method are applied: Respondents need to drag the response options towards the rating scale items (“drag-response”), or in reverse, the rating scale items towards the response options (“drag-item”). Both drag-and-drop scales aim at encouraging respondents to process rating scale items more attentively and carefully instead of simply relying on cognitive shortcuts.

Methods & Data: In two randomized field experimental Web surveys conducted among university applicants (n=5,977 and n=7,395), various between-subjects designs were implemented to assess the effectiveness of the drag-response and drag-item scale in preventing the respondents’ susceptibility to cognitive shortcuts compared to a standard grid question in terms of different systematic response tendencies commonly encountered with rating scales such as nondifferentiation and primacy effects. Furthermore, item missing data and response times were examined.

Results: Findings revealed that the quality of answers to rating scales may profit from the respondents’ higher attentiveness and carefulness in both drag-and-drop scales which is reflected in a decreased susceptibility to systematic response tendencies. At the same time, however, results also showed that both drag-and-drop scales entail a higher extent of respondent burden compared to conventional radio button scales as indicated by increases in item missing data and longer response times.

Added Value: This study provides a comprehensive examination of the potentials and limitations of new drag-and-drop procedures as an interactive data input method for rating scales in Web surveys. In addition, findings contribute to a better understanding of the cognitive processing of rating scales in Web surveys.